Add analyte-trend, MCL-exceedance, and monitoring-recency products#90
Merged
jirhiker merged 6 commits intoJun 28, 2026
Merged
Conversation
Three new data products built on the per-source asset graph: - ogc_analyte_trend: per-well analyte concentration trend (Mann-Kendall + Theil-Sen, daily mean). One product per analyte; seeds nm_arsenic_trend and nm_nitrate_trend. - ogc_mcl_exceedance (nm_mcl_exceedance): one feature per well flagging drinking-water MCL exceedances. Thresholds read at run time from gs://<bucket>/config/mcl.json (source of truth); see mcl.sample.json. - ogc_monitoring_recency (nm_monitoring_recency): one feature per well with last-observation date, days_since_last, and active/stale status (water levels, stale > 365d). Implementation: - Generalize the trend dumper: dump_waterlevel_trend_collection -> dump_trend_collection(slope_units, reducer, method, parameter_name); _daily_min_series -> _daily_series(reducer min|max|mean). slope_ft_per_year -> slope_per_year + slope_units. - New dumpers dump_mcl_exceedance_collection (pivot + threshold compare) and dump_monitoring_recency_collection. - GCSResource.read_json for the MCL file; die_config treats MCL as summary mode; definitions registers the three output types (each gets a job + schedule). Offline tests cover all three. Run nm_mcl_exceedance only after uploading config/mcl.json to the products bucket. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Your pull request is automatically being deployed to Dagster Cloud.
|
Replace the sample with the real EPA-sourced MCL file. Values in mg/L: - arsenic 0.01, nitrate(as N) 10, fluoride 4.0, uranium 0.03 (primary) - chloride 250, sulfate 250, tds 500 (secondary) pH (6.5-8.5) omitted (a range, not a single MCL). Provenance + EPA source URLs recorded in the file. Add uranium to the nm_mcl_exceedance analyte list. Upload this file to gs://<products_bucket>/config/mcl.json. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a _schema block (explains every field), structured _source with EPA URLs + retrieved date, an _omitted note (pH is a range), and per-analyte units/basis/label/note. The product reads only mcl/type per analyte; _-prefixed keys and extra fields are ignored, and the whole dict travels into the output collection's mcl_thresholds as provenance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The exceedance test is a direct magnitude comparison, so MCL and value must share units and basis. Document the nitrate pitfall (EPA MCL is as N; data may be as NO3, ~4.43x) at the comparison site. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ogc_features.py had grown to ~628 lines mixing three concerns; the statistics cluster is the one that isn't serialization. Move the daily aggregation, qualification gate, Mann-Kendall + Theil-Sen test, the thresholds, and the method-description text into backend/trend_stats.py (pure analysis, lazily importing scipy/pymannkendall). ogc_features re-exports them so importers and dump_trend_collection's default arg keep working. ogc_features 628 -> 506 lines (serialization only); trend_stats 143. Add direct unit tests for the extracted module. 27 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dg kept disappearing from the project venv on env re-resolves because it was never declared, breaking the AGENTS.md-recommended `dg check defs`. Declare it so `uv run dg ...` always works. Dev/CLI only — not in the serverless requirements.txt, so the deploy PEX is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jirhiker
force-pushed
the
feature/trend-mcl-recency-products
branch
from
1f35c31 to
ec1607b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #89 (uses the generalized trend engine + source_datastream_link). Base will retarget to main once #89 merges.
What
Three new data products on the per-source asset graph:
ogc_analyte_trend— per-well analyte concentration trend (Mann-Kendall + Theil-Sen, daily mean). One product per analyte; seedsnm_arsenic_trend+nm_nitrate_trend.ogc_mcl_exceedance(nm_mcl_exceedance) — one feature per well flagging drinking-water MCL exceedances. Thresholds read at run time fromgs://<products_bucket>/config/mcl.json(source of truth). Per analyte: value,_mcl,_mcl_type,_exceeds; plusany_exceedance,exceedance_count,exceeded_analytes.ogc_monitoring_recency(nm_monitoring_recency) — one feature per well:last_observation_datetime,days_since_last,status(active/stale atstale_days, default 365). Surfaces dead/lagging monitoring points.Implementation
dump_waterlevel_trend_collection→dump_trend_collection(slope_units, reducer, method, parameter_name);_daily_min_series→_daily_series(reducer = min|max|mean). Output fieldslope_ft_per_year→slope_per_year+slope_units. WL trend now calls it withreducer="min", slope_units="ft/year"; analyte trend withreducer="mean", slope_units="mg/L/year".dump_mcl_exceedance_collection(well pivot + threshold compare) anddump_monitoring_recency_collection.GCSResource.read_jsonfor the MCL file.die_configtreatsogc_mcl_exceedanceas summary mode (latest value per analyte).definitionsregisters the three new output types — each auto-gets a job + schedule.products.yaml: 4 product entries.mcl.sample.jsondocuments the schema (real file lives in GCS).Verification
nm_arsenic_trend,nm_nitrate_trend,nm_mcl_exceedance,nm_monitoring_recency).Before running
nm_mcl_exceedanceUpload
config/mcl.jsonto the products bucket (schema =orchestration/config/mcl.sample.json), values in mg/L.🤖 Generated with Claude Code